Based on the theories of rotor dynamics and gear meshing, the finite element model of a gear-shaft system was established according to a certain gear transmission system of a steam unit. The bending and torsional modes of the single gear wheel and coupling gear-shaft system were calculated and compared respectively. The effect of gear stiffness on the critical speeds of the coupling gear-shaft system was studied. Moreover, the imbalance responses of the single and coupling system were employed. The calculated results offer some academic reference of gear system design.